Method, computer program product, and system for providing a sensor-based environment

ABSTRACT

Method, computer program product, and system to provide an extended vision within an environment having a plurality of items, where the extended vision is based on a field of view of a person determined using a first visual sensor, and is further based on at least a second visual sensor disposed within the environment. Image information from the first and second visual sensors is associated to produce combined image information. Selected portions of the combined image information are displayed based on input provided through a user interface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit of U.S. provisional patent applicationSer. No. 62/064,323, filed Oct. 15, 2014, entitled “Integrated ShoppingEnvironment,” which is herein incorporated by reference.

BACKGROUND

The present disclosure relates to a sensor-based environment, and morespecifically, techniques for providing an extended vision based on afield of view of a person within the environment and one or more visualsensors disposed throughout the environment.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 illustrates an exemplary environment including a plurality ofitems, according to one embodiment.

FIG. 2 illustrates example limitations of vision for a person in anenvironment, according to one embodiment.

FIG. 3 illustrates an exemplary system for providing an extended visionwithin an environment based on a determined field of view for a person,according to one embodiment.

FIG. 4 illustrates the operation of an exemplary system for providing anextended vision within an environment based on a field of view for aperson, according to one embodiment.

FIG. 5 illustrates example devices for determining a field of view of aperson, according to one embodiment.

FIGS. 6A and 6B illustrate image information acquired by a plurality ofvisual sensors, according to one embodiment.

FIG. 7 illustrates an example display of combined image information,according to one embodiment.

FIG. 8 illustrates a method of providing an extended vision within anenvironment having a plurality of items for selection, according to oneembodiment.

FIG. 9 illustrates a method of producing combined image information toaddress deadspace areas in the environment, according to one embodiment.

FIG. 10 illustrates a method of producing combined image informationbased on overlap areas in the environment, according to one embodiment.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures. It is contemplated that elements disclosed in oneembodiment may be beneficially utilized on other embodiments withoutspecific recitation. The illustrations referred to here should not beunderstood as being drawn to scale unless specifically noted. Also, thedrawings are often simplified and details or components omitted forclarity of presentation and explanation. The drawings and discussionserve to explain principles discussed below, where like designationsdenote like elements.

DETAILED DESCRIPTION

Aspects of the current disclosure relate to an integrated environmentcapable of providing a personalized, automated, and adaptive experiencefor a person within the environment. A number of different sensordevices may be employed within the environment, and networked withvarious computing devices such as point-of-sale (POS) terminals, digitalsignage, servers, and mobile or handheld computing devices to provide aseamless integration of mobile technologies and e-commerce intotraditional experiences.

Embodiments disclosed herein are generally directed to providing anextended vision within an environment having a plurality of items thatare available for selection. The extended vision is based on adetermined field of view of a person, such as a customer or employee,having a first computing device coupled with a first visual sensor. Theenvironment is also associated with at least a second computing devicecoupled with one or more second visual sensors, which may be disposedthroughout the environment. The image information from the first andsecond visual sensors is associated to produce combined imageinformation, which may be displayed on a display device and that may betraversed using a user interface (UI). In some embodiments, the personhaving the first computing device may view the displayed combined imageinformation, scrolling or otherwise navigating beyond their naturalfield of view to observe items located elsewhere in the environment. Insome embodiments, the combined image information may be available forviewing or reviewing by employees in the course of performing securityfunctions. While many concepts and terms are generally discussed withinthe context of a shopping environment, such as a retail store, it iscontemplated that the techniques disclosed herein may be applied toother environments (some non-limiting examples include libraries,museums, classrooms, hospitals, etc.) to provide an adaptive experiencefor persons included therein.

FIG. 1 illustrates an exemplary environment including a plurality ofitems, according to one embodiment. The environment 100 includes aplurality of sensor modules 105 disposed in the ceiling 110 of theenvironment. The sensor modules 105 may each include one or more typesof sensors, such as video sensors (e.g., cameras), audio sensors (e.g.,microphones), and so forth. Sensor modules 105 may also includeactuating devices for providing a desired position and/or orientation ofthe included sensor(s). Generally, the sensor modules or individualsensors may be disposed at any suitable location within the environment100. Some non-limiting examples of alternative locations include below,within, or above a floor 115 of the environment, within other structuralcomponents of the environment 100 such as a shelving unit 120 or walls,and so forth. In some embodiments, sensors may be disposed on, within,or near item display areas such as the shelving unit 120. The sensorsmay be oriented toward expected locations of personal interactions withitems in order to acquire better data about the person's interactions,such as determining the person's field of view relative to certainitems, updating a virtual cart or transaction record for the person'stransaction in the environment, and so forth.

Environment 100 also includes a number of computer-based terminals (orkiosks) 125. Generally, terminals 125 may be configured for performingcustomer checkout and/or other functions, such as providing informationto a customer or employee. Each terminal 125 may each include a discretecomputing device or portions of a computing system, and may includevarious I/O devices, such as visual displays, audio speakers, cameras,microphones, etc. for interacting with various persons such as customersand/or employees. In some embodiments, a person 130 in the environmentmay have a mobile computing device, such as a smartphone 135, thatcommunicatively couples with the terminal 125 for completing a checkouttransaction. For example, the person's smartphone 135 may includepayment information, identification information, etc. that facilitatecompletion of the checkout transaction. In one embodiment, the mobilecomputing device may execute a store application that connects with thecomputing system of the environment (e.g., to store servers or othercomputing devices through the Internet). In one embodiment, the mobilecomputing device may be directly connected with kiosk 125 throughwireless networks established within the environment (e.g., over Wi-Fior Bluetooth). In one embodiment, the mobile computing device may couplewith the kiosk 125 when brought within range, e.g., using Bluetooth ornear-field communication (NFC).

Environment 100 also includes one or more shelving units 120 havingshelves 140 that support various store items 145. Though not shown,multiple shelving units 120 may be disposed in a particular arrangementin the environment 100, with the space between adjacent shelving unitsforming aisles through which customers and employees may travel. Forexample, customers may navigate the aisles and/or approach the shelvingunits 120 to view items 145 included therein, to handle the items, toselect the items, etc. In another example, employees may navigate theaisles and/or approach the shelving units 120 to view stock levels ofthe items 145, to determine out-of-place items, etc. In someembodiments, shelving units 120 may include visual sensors or othersensor devices or I/O devices. The sensors or devices may couple withthe person's smartphone 135 and/or other networked computing devices(including terminals 125 and/or servers) that are associated with theenvironment 100. For example, the front portions 150 of shelves 140 mayinclude video sensors oriented outward from the shelving unit 120 (i.e.,toward the aisle) to acquire image information for a person'sinteractions with items 145 on the shelving unit 120, with the imageinformation provided to back-end servers for storage and/or analysis. Insome cases, some or all of the image information may also be accessibleby a person's mobile computing device. In some embodiments, portions ofthe shelving unit 120 (such as the front portions 150 of shelves 140)may include indicator lights or other visual display devices or audiooutput devices that are able to communicate with a person.

During an exemplary transaction in the environment, the person 130 mayhave a shopping receptacle in which the person places items after theyare selected for purchase. Examples of shopping receptacles includeshopping carts, baskets, or other containers that may be carried orotherwise transported by the person during the transaction. Uponcompletion of the transaction—for example, the person has selected allof the desired items—the person may approach one of the terminals 125 ora designated checkout area to perform a checkout transaction.

During a transaction, a person at a particular location may desire toview different areas within the environment, but is unable to do so forone or more reasons. In some cases, certain items in the environment maybe disposed on relatively high or low shelving, but the person who seeksto view the items is not naturally able to do so—for example, the personmay be too short to view items on high shelving or not capable ofstooping or bending to view items on low shelving. In some cases, aperson may seek to view items that are located outside their naturalfield of view at their location—for example, the items may be locatedfurther down a same aisle on the same or different shelving, or perhapswithin a different aisle or area altogether.

Providing an extended vision for the person using image informationacquired from visual sensors may be advantageous across a number offunctions ordinarily performed within the environment. In the examplediscussed above, a customer while shopping may use the extended visionto view out-of-sight items or items located remotely from the customer'slocation. In another example, an employee may use the extended vision toperform inventory checking for out-of-sight items or remotely locateditems. In another example, an employee or administrator of theenvironment may use the extended vision as part of a security function,monitoring different areas of the environment. In another example, theadditional image information as part of the extended vision—even ifspatially overlapping with other image information—may be used tofurther enhance gesture and item identification.

FIG. 2 illustrates example visual limitations for a person in anenvironment, according to one embodiment. Scene 200 illustrates a sideview depiction of a person 205 viewing the front of a shelving unit 120.Items that may normally be displayed on shelving unit 120 are notdepicted here for clarity. The shelving unit 120 includes severalstructural members, such as a back member 215 and one or more shelves140 ₁₋₄ coupled therewith. The back member 215 may have a planar shapeand extend along a length of the shelves 140 (as shown, the direction ofthe length may extend into and out of the page). In some cases, the backmember may be wall-mounted or base-supported as part of a standaloneshelving unit (e.g., a gondola unit). The shelves 140 may be coupleddirectly with the back member 215, e.g., using hooks, tabs, or othercomplementary fittings, and/or fasteners. The shelves may alternativelybe coupled with one or more braces or brackets 220, which in turn arecoupled with the back member 215.

The person 205 is associated with a field of view 210. Fields of viewmay differ based on the characteristics of the person, such as theirheight, as well as based on their relative position and orientation. Forexample, a person standing on his or her tiptoes may have a differentfield of view than the same person standing flatfooted, kneeling,sitting, etc. Additionally, a person standing nearer to the shelvingunit 120 may have a narrower field of view than if the person werestanding further away. As shown, the person 205 is able to observe itemson the shelves 140 ₂, 140 ₃, and items positioned towards the front ofshelf 140 ₄. However, the person 205 is unable to observe items includedin one or more areas 225 ₁, 225 ₂ due to a visual occlusion that iscaused by the shelves 140 ₁, 140 ₃.

Scene 240 provides a top view that generally corresponds to the sideview illustrated in scene 200. Here, the person 205 is viewing a firstshelving unit 120A; a second shelving unit 120B is spatially separatedfrom the first shelving unit 120A by an aisle 250. The shelving unit120A includes a back member 215A and shelves 140A₁ and 245A₁ on opposingsides of the back member 215A. As shown, the person 205 has a field ofview 210 that includes a portion of shelf 140A₁. However, based on theircurrent location, the person 205 may be unable to see items included inan area 255 on the shelf 140A₁. Additionally, the back member 215A mayobscure the person's vision such that the person 205 cannot effectivelyview items on other shelf 245A₁ or on shelving unit 120B through theback member 215A.

FIG. 3 illustrates an exemplary system for providing an extended visionwithin an environment based on a determined field of view for a person,according to one embodiment. The system 300 includes a number ofcomponents that are disposed within the environment 100. The system mayalso include components that are outside the environment —for example, aserver 365 may be located remotely or proximately disposed to theenvironment (such as within a back room in the same building that is notaccessible by customers).

Components within the environment include one or more sensors 305 ofvarious types, such as visual sensors 310, audio sensors 315, and weightsensors 320. The sensors 305 may also include other sensors 325 (whichmay also include infrared sensors, ultrasonic sensors, etc.). Thesensors 305 generally include any sensors that are capable of providingmeaningful information about customer interactions with the environment,e.g., location sensors, weight sensors, eye gaze tracking sensors. Thesensors 305 may be discrete sensor devices deployed throughout theenvironment 100 in fixed and/or movable locations. Sensors 305 may bestatically included in walls, floors, ceilings, displays, or othernon-sensor devices, or may be included in shopping receptacles capableof being transported through the environment. In one embodiment, sensors305 may include adjustable-position sensor devices, such as motorizedcameras (i.e., an example of visual sensors 310) attached to a rail,wire, or frame. In one embodiment, sensors 305 may be included on one ormore unmanned vehicles configured to travel through some or all of theenvironment 100, such as unmanned ground vehicles (UGVs) or unmannedaerial vehicles (UAVs or “drones”). Sensors 305 may also include sensordevices that are included in computing devices associated with theenvironment 100, such as personal devices 330, employee devices 335, andterminals 340. In some cases, the computing devices (or the componentsensor devices) may be implemented as body-worn or carried devices.

Personal devices 330 and employee devices 335 may each include passiveor actively-powered devices capable of communicating with at least oneof the networked devices of system 300. One example of a passive device(which may be worn or carried) is a NFC tag. Active devices may includemobile computing devices, such as smartphones or tablets, or wearabledevices such as a Google Glass™interactive eyepiece (Glass is atrademark of Google Inc.). The personal devices 330 generally denotesownership or possession of the devices by customers within theenvironment 100, while the employee devices 335 denotes ownership orpossession by the retailer or other administrator of the environment100. In some cases, employee devices 335 may be carried by employees andused in the course of their employment. Personal devices 330 andemployee devices 335 may execute applications or other program code thatgenerally enables various functions and features accessible using server365 and/or other networked computing devices. In some embodiments,sensor devices that are included with the personal devices 330 oremployee devices 335 may be included in the sensors 305.

System 300 includes a plurality of terminals 340 within the environment100. Terminals 340 generally include any structure that is capable ofreceiving input from and/or producing output to people (e.g., customers,employees) within the environment 100. The terminals 340 may includecomputing systems, portions of computing systems, or devicescontrollable by computing systems. In one example, a terminal 340 mayinclude a computing device that is communicatively coupled with a visualdisplay and audio speaker(s), as well as being communicatively coupledwith one or more input devices. In another example, a terminal 340 mayinclude a visual display and associated driver hardware, but a computingdevice coupled to the terminal and providing data for display isdisposed separately from the terminal. In some embodiments, terminals340 may be implemented as standalone devices, such as a kiosk disposedon the store floor or monolithic device disposed on a shelf or platform.In some embodiments, terminals 340 may be integrated partially or whollywith other components of the environment 100, such as input or outputdevices included with shelving or other structural components in theenvironment (e.g., components used for product display or storage). Insome embodiments, terminals 340 may be modular and may be easilyattachable and detachable to elements of the environment 100, such asthe structural components.

Generally, terminals 340 may be distributed throughout the environment100 and may enhance various phases of the person's transactions withinthe environment. For example, terminals 340 may include digital signage(i.e., included as an example of other terminals 355) disposedthroughout the environment, such as included in or near aisles, endcaps,displays, and/or shelving in the environment. A person during atransaction may view and/or interact with the digital signage as he orshe moves throughout the environment. The digital signage may beincluded in a static display or may be movable, such as includingdigital signage within a shopping receptacle. Terminals 340 may alsoinclude one or more types of terminals usable for completing checkouttransactions, such as employee-manned POS terminals 345 andself-checkout terminals 350. In some cases, the terminals 340 thatprovide checkout functionality may be disposed within a designatedcheckout area within the environment 100.

In some embodiments, terminals 340 may provide an integratedfunctionality. For example, terminals 340 may function in a first modeas digital signage, and when engaged by a person (i.e., receiving inputfrom the person), the terminals function in a second mode as aself-checkout terminal or other type of terminal.

Server 365 generally includes processor(s), memory, and communicationscapabilities and may perform various computing tasks to support theoperation of the environment 100. Server 365 may communicate usingvarious wired and/or wireless communications methods with terminals 340,sensors 305, and with other networked devices such as personal devices330 and employee devices 335. Server 365 generally executes computerprogram code in which input data is received from networked devices, theinput data is processed and/or stored by the servers, and output data isprovided to networked devices for operation of the environment 100.

Network 360 may include one or more networks of various types, includinga local area or local access network (LAN), a general wide area network(WAN), and/or a public network (e.g., the Internet). In one embodiment,various networked computing devices of the system 300 are interconnectedusing a LAN, and one or more computing devices (e.g., server 365,personal devices 330) include connections to the Internet.

FIG. 4 is a block diagram illustrating operation of a system forproviding an extended vision within an environment based on a determinedfield of view for a person, according to one embodiment. Generally, thearrangement 400 represents one possible implementation of the system 300that is suitable for use within the environment 100.

Arrangement 400 includes the server 365, as well as one or more personaldevices 330 _(1,2) associated with customers and one or more employeedevices 335. Alternatively, the arrangement 400 may include a differentcomputing device that is generally associated with the environment, suchas a terminal or kiosk. In some embodiments, the personal devices 330and employee devices 335 are mobile computing devices, such assmartphones, tablets, or other worn or carried computing devices.Collectively, the personal devices 330 and employee devices 335 may bereferred to as “mobile devices.” Each of the computing devices ofarrangement 400 (i.e., server 365, personal devices 330, employeedevices 335) includes respective processors 410, 465, as well as memory415, 470 and input/output (I/O) 450, 475.

Through I/O 450, the server 365 is coupled with a user interface (UI)452, a display 454, one or more visual sensors 456, and optionally otherinput or output devices 458. Examples of other input devices includeother types of sensors, as well as standard computer input devices suchas a keyboard, trackpad, touchscreen, etc. Examples of other outputdevices include other sensory output devices (e.g., audio, haptic,etc.). Although not shown, I/O 450 includes networking connections toenable communication with the mobile devices and/or with other networks,such as the Internet. Similarly, the mobile devices may include a visualsensor 476, other sensors 478, a display 480, and a UI 482.

Memory 415 may store various image information 425 that is acquiredusing the visual sensors 456. The image information 425 may be in anysuitable form, such as image files, frames of video files, etc. Like theserver 365, the visual sensors 456 may also be associated with theenvironment. In some embodiments, the visual sensors 456 may be deployedin a relatively static arrangement throughout the environment (e.g.,installed overhead cameras). In some embodiments, some or all of thevisual sensors 456 may be movable, such as in reconfigurable productdisplays. In some embodiments, the visual sensors 456 are actuable, andmay be steered or displaced (e.g., such as aerial drones).

In some embodiments, the image information 425 in memory 415 alsoincludes image information 474 that is acquired using other visualsensors, such as visual sensors 476 included with mobile devices. Theimage information 474 may be acquired during use of the mobile devicesin the environment. In other embodiments, the image information 425 and474 are stored in separate memories (memory 415, 470). To generate thecombined image information 435 using both image information 425 andimage information 474, the server 365 must first be associated with themobile devices. In one embodiment, associating the computing devices maybe performed wirelessly, and the process may be initiated by executingan application 472 on the personal device 330 ₁.

The memory 415 may also store other data related to the environment andassociated persons, such as item information 430. Item information 430may relate to various aspects of the items, such as physicalcharacteristics, reference images, pricing, current inventory levels,and so forth. The item information 430 may also include location data432 that indicates relative placement of the items within theenvironment. In one embodiment, the location data 432 may be associatedwith one or more planograms that generally show a planned arrangementfor item placement.

The memory 415 includes one or more functional modules that may beexecuted using the processors 410, such as an image processing module420. Image processing module 420 generally operates on image information425 and/or image information 474 to produce the combined imageinformation 435. The image processing module 420 may include sub-modulesfor performing item identification 421 and image association 422. Itemidentification 421 may be performed according to any suitablealgorithms, as would be known to the person of ordinary skill in theart. Likewise, image association 422 may be performed according to anysuitable algorithms. In one embodiment, image association 422 performsimage stitching with image information 425, 474. In some cases, imageassociation 422 may beneficially use item identification 421 to matchidentified items across image information provided by different ones ofthe visual sensors 456, 476. Image processing module 420 may alsoinclude any other desired image processing functions, such as performingspatial transformations for different image information, sharpening orsmoothing the image information, etc.

The combined image information 435 includes image information fromvarious visual sensors that is associated to form a more or lesscontinuous image of portions of the environment. The continuous imagemay be traversed or navigated, and portions displayed to a person usingserver 365 or associated computing devices (personal devices 330,employee devices 335). In some embodiments, the combined imageinformation 435 represents image information 425 that is acquired byenvironment visual sensors (i.e., visual sensors 456) and visual sensors474 for individual persons, customers, etc. The combined imageinformation 435 may be represented as a 2D or 3D representation of theentire environment, or of selected portions thereof. In one example, theentire environment may be modeled in 3D. In another example, thegenerated combined image information 435 may correspond to 2D frontviews of shelving units within the environment. In one embodiment, thecombined image information 435 may spatially correspond to planogramsfor the environment.

The combined image information 435 may be presented to a viewer using adisplay device, such as a display 454 coupled with the server 365, orthe display 480 of one of the mobile devices. In some embodiments, thecombined image information 435 is able to be navigated or traversed viaa user interface (UI) for the corresponding computing device. As will befurther discussed, the display of combined image information 435provides an extended vision for the viewer that may be useful forovercoming limitations of natural vision (e.g., out-of-sight areas) orgaps in imagery provided by an arrangement of visual sensors.

Determining the combined image information 435 may reveal one or moredeadspace areas 436—that is, areas of the environment that the imageinformation acquired from the visual sensors 456, 476 does not include.Generally, deadspace areas are typically dictated by the arrangement ofthe visual sensors in the environment (e.g., disposition andorientation). In the case of visual sensors having a fixed arrangement,the deadspace areas are not easily overcome, often requiring an employeeor technician to physically rearrange the visual sensors. In some cases,the visual sensors 456 may be controlled by the server 365, such as bysteering or displacing the visual sensors, which may reduce the sizingand/or number of deadspace areas. In some embodiments, the visualsensors 476 of the mobile computing devices may be advantageouslyemployed to improve the overall coverage of the combined imageinformation 435 and to overcome limitations of the arrangement of visualsensors 456.

In some embodiments, the server 365 may transmit a prompt to a person(e.g., a customer or employee) associated with a particular mobiledevice in order to acquire image information corresponding to thedeadspace areas 436. In some cases, the server 365 may userecommendation module 448 and/or advertising module 449 to encourage theperson's cooperation. For example, the server 365 may determine one ormore items that are located within or adjacent to a particular deadspacearea 436, e.g., referencing location data 432 and/or nearby imageinformation, and suggest the one or more items to the mobile deviceusing recommendation module 448. In another example, the server 365could overtly request the person's assistance, e.g., requesting that theperson go to the area to acquire image information via their mobiledevice. In some embodiments, the person's assistance may be incentivizedusing advertising or offering customized discounts, rewards points, etc.

The combined image information 435 in many cases will also includeoverlap areas 438, which represent areas of the environment included inimage information that is acquired by two or more visual sensors 456.For various reasons, it may be beneficial to assess the duplicativeimage information and to determine a relative priority of the imageinformation for use in the combined image information 435. Usingprioritized image information may result in the construction of a bettercombined image. For example, image information provided by one visualsensor may have a higher resolution (i.e., generally higher priority),or image information from one visual sensor may be relatively distorteddue to the overlap area 438 occurring near the periphery of the visualsensor (i.e., generally lower priority), etc. Of course, any number ofother factors may be considered when prioritizing image information. Forexample, even though a visual sensor provides lower resolution imagery,the sensor could receive a higher priority where a small size of theimage information is desirable.

Priority information 440 may be used to designate the relative priorityof image information corresponding to overlap areas 438. In some cases,the priority may serve as a measure of the relative quality of the imageinformation. As shown, two classes of priority are included—primary 442and auxiliary 444—although in other embodiments different numbers oflevels may be used. In one embodiment, one image information (i.e.,corresponding to a particular visual sensor) for an overlap area 438 isclassified as primary, and all other image information (i.e., from othervisual sensors) is classified as auxiliary. In some cases, the primaryimage information is used to construct the combined image information,instead of using the auxiliary image information. In some cases,auxiliary image information may be used in the absence of primary imageinformation (e.g., the corresponding visual sensor malfunctions). Forexample, the server 365 may select auxiliary image informationcorresponding to the next-largest number of identified items.

In some embodiments, the relative priority of image information may bedetermined based on numbers of identified items. In one embodiment, theimage information corresponding to a larger number of identified itemsis designated as primary image information for the overlap area, whileimage information having lesser numbers of items is designated asauxiliary. In one example, comparing numbers of identified items isbased on the number of identified items corresponding specifically tothe overlap area. In another example, the comparison is based on thenumber of items identified within the entire image information. In someembodiments, the number of identified items may be compared with athreshold value 446. If none of the numbers of identified items for thedifferent image information meets the threshold value, the server 365may determine that the image information is inadequate for the overlaparea 438. Accordingly, the server 365 may prompt a person in order toobtain new image information for the overlap area, such as by providinga recommendation or request to a person.

FIG. 5 illustrates example devices for determining a field of view of aperson, according to one embodiment. By using the visual sensors to makea computer-based determination of the person's field of view, combinedimage information may be seamlessly presented to the person through anassociated display device. The combined image information, whendisplayed to the person, may be navigated or traversed to overcome theperson's natural visual limitations. Additionally, the image informationacquired from the visual sensors may be used to improve the coverageand/or quality of the combined image information when displayed forother persons.

Scene 560 depicts a person 502 wearing a wearable computing device 565on her head 504. The wearable computing device 565 includes supportmembers such as a band 566 coupled with nosepieces 568. Although notshown, band 566 may be coupled with the person's head at additionallocations, e.g., using compression properties of the band, earpieces,etc. The band 566 supports other structural components, such as ahousing 570 that may include the computing device and associatedhardware and connections. The housing 570 includes a forward-lookingvisual sensor 571 oriented through an opening formed in the housing.During operation, the forward-looking visual sensor 571 generallyrepresents the person's field of view. The housing 570 is also coupledwith a display 572. In some cases, the display 572 may be transparentand provide information as a visual overlay meant to complement theperson's natural field of view.

Scene 580 depicts a person 502 holding a mobile computing device 585 inher hand 503. The mobile computing device 585 includes a display 586 ona first surface and a visual sensor 587 on a second surface (not shown;opposite the first surface). The person 502 adjusts the orientation ofthe mobile computing device to reorient the visual sensor 587, which maybe provided as immediate feedback using the display 586. To acquirefield of view information for the person, the person 502 may “aim” thevisual sensor 587 at a desired portion of the environment by raising,lowering, or otherwise manipulating the mobile computing device 585.

FIGS. 6A and 6B illustrate image information acquired by a plurality ofvisual sensors, according to one embodiment. Scene 600 depicts a frontview of a shelving unit 610 having a number of shelves 615 ₁₋₅. In oneembodiment, the scene may correspond to a planogram graphic for theshelving unit. Each shelf 615 supports and displays a number ofdifferent items (no reference numbers provided). Visual sensors disposedwithin the environment acquire image information corresponding toportions of the scene 600. For example, a first visual sensor acquiresimage information 620A while a second visual sensor acquires imageinformation 620B. For simplicity, the image information is shown asrectangular; however, persons of ordinary skill will recognize that thevisual sensors may acquire varying shapes and sizes of imageinformation. Further, the image information may be spatially transformedto produce the front view depicted in scene 600. When processed togenerate the combined image information, the areas of image information620A, 620B define an overlap area 630. The areas of image information620A, 620B also define deadspace areas 635A, 635B having no visualsensor coverage.

Scene 650 depicts a similar view as scene 600 and includes overlaidfield of view information for a person. For example, the field of viewinformation may be based on image information acquired from aforward-looking visual sensor of a worn computing device, or a visualsensor included in a carried computing device.

Field of view information 655A corresponds to image information acquiredat a first time and may represent the person viewing items on shelves615 ₄ and/or 615 ₅. As shown, the field of view area 655A completelyoverlaps the image information 620B and partly overlaps with imageinformation 620A. In one embodiment, the image information correspondingto field of view area 655A is not stored for use with the combined imageinformation, as the field of view area is already adequately representedby image information 620A, 620B. In another embodiment, the imageinformation is stored consistent with priority criteria. For example, ifthe image information for field of view area 655A is of higher qualitythan the image information 620A, 620B, the field of view information maybe stored.

At a second time, the person may have field of view area 655B whenviewing items on top shelf 615 ₁. In one embodiment, the imageinformation corresponding to field of view area 655B will be stored foruse with the combined image information, as the field of view area atleast partly overlaps with one or more deadspace areas 635B.

In one embodiment, the combined image information is updatedperiodically by resampling the image information from the visualsensors. In some cases, the updating may be performed substantiallycontinuously (e.g., a video feed). However, the image informationcorresponding to a person's field of view may be transitory, as theperson will likely leave the area or at least look elsewhere. Thus, insome embodiments, the image information corresponding to the person'sfield of view may be acquired and stored as a still image and, asappropriate, included in the updated combined image information. In oneembodiment, the image information corresponding to the person's field ofview is updated when a person next acquires image informationcorresponding to the same deadspace area.

FIG. 7 illustrates an example display of combined image information,according to one embodiment. Scene 700 depicts combined imageinformation 705. The combined image information 705 may have anysuitable format, such as video, image, or graphics files. In oneembodiment, the combined image information 705 may correspond to aplanogram of a shelving unit in the environment. The combined imageinformation 705 includes regions of image information acquired byvarious visual sensors, such as image information 710, 712, 714, 716,and 725. Of course, while displayed as rectangles for simplicity, theimage information may have any shape. The combined image information 705may contain overlap areas, as well as one or more deadspace areas 720A,720B, 720C.

Selected portions of the combined image information 705 may bedisplayed, for example, using a display for a mobile computing device orother computing device. Generally, movement of the display area 730 maybe used to provide an extended vision for the person using the computingdevice. In one example, the person (e.g., a customer or employee) mayhave a current field of view corresponding to the image information 725,but may wish to view items on a top shelf, to browse items the sameshelving unit further down the aisle, to view items on another aisle,etc. In another example, the person (e.g., an employee) may wish tonavigate the combined image information to check inventory levels,provide security, and so forth. Based on inputs provided through a UI,the display area 730 may be moved relative to the combined imageinformation 705 (i.e., navigating) to display desired portions of thecombined image information. In some embodiments, the display may presentvisual UI elements, such as arrows 735A-D, in addition to the portion ofthe combined image information 705 included in the display area 730.When pressed or otherwise selected, the arrows 735 cause the displayarea to be adjusted in the corresponding direction. In otherembodiments, the computing device may include a touchscreen that is ableto accept gestures, such as finger swiping, pinching motions, etc. tonavigate the combined image information 705.

In some embodiments, the navigation functions provided by the UI may beused to navigate across a substantially continuous combined imageinformation (e.g., a 3D modeling of the environment). For example, theperson could navigate along areas that are ordinarily accessible tocustomers (such as along aisles) as well as areas not normallyaccessible—the person could navigate between aisles, say, by navigatingabove the top portions of shelving units. In some embodiments, thenavigation functions may be performed across discrete portions of thecombined image information. In one example, if combined imageinformation includes discrete 2D representations of each shelving unit,the UI may allow the viewer to select the particular shelving unit (oraisle), and then navigate along the selected shelving unit. In anotherexample, the navigation between discrete portions may be seamless (e.g.,navigating past an end of one shelving unit causes a next shelving unitto be displayed).

Deadspace areas 720 may be represented within the combined imageinformation 705 in any suitable manner. For example, deadspace areas mayremain blank, may be grayed out, or may use blending from nearby imageinformation. In one example, deadspace area 720C could include agradient or other transitional graphics from any of image information710, 712, 714, 716 that define the area. In another example, the imageinformation for deadspace areas may be estimated, such as by copying aportion of adjacent image information. In another example, deadspaceareas may include other types of graphical substitution, such asinserting a corresponding portion of planogram graphics. As discussedabove, the deadspace areas 720 may be addressed by prompting a person toacquire image information for the area via their (mobile) computingdevice, and incentivized using advertising or offering customizeddiscounts, rewards points, etc.

FIG. 8 illustrates a method of providing an extended vision within anenvironment having a plurality of items available for selection,according to one embodiment. Generally, method 800 may be usedconsistent with the systems and environments described herein. Method800 begins at block 805, where a first computing device is associatedwith a second computing device. In one embodiment, the first computingdevice is associated with a person, such as a mobile computing devicethat is worn or carried by the person, and the second computing deviceis associated with the environment, such as a back-end store server. Inone embodiment, associating the first and second computing devices mayinclude opening an application on the first computing device for theenvironment. In one embodiment, associating the first and secondcomputing devices may be performed upon detecting the presence of thefirst computing device within the environment.

At block 815, first image information from a first visual sensor isanalyzed to determine a field of view of the person. The first visualsensor may be included in the first computing device associated with theperson. In some embodiments, the first visual sensor is aforward-looking visual sensor of a body-worn computing device. In otherembodiments, the first visual sensor is included in a smartphone orother mobile computing device.

At block 825, second image information from a second visual sensor isanalyzed. In one embodiment, the second visual sensor is coupled withthe second (environment) computing device, and includes one or morevisual sensors that are distinct from the first visual sensor. In oneembodiment, the one or more second visual sensors are arrangedthroughout the environment.

At block 835, and based on the determined field of view, the first andsecond image information is associated to produce a combined imageinformation. In one embodiment, the combined image information may begenerated and stored using the second (environment) computing device.The association of image information may be performed according to anyfeasible algorithms. In one embodiment, the image information may bestitched to generate the combined image information. In someembodiments, other image processing is performed to the first and/orsecond image information, such as spatial transformation and/or imageenhancement.

At block 845, deadspace and/or overlap areas are optionally determinedbased on the combined image information. Deadspace areas correspond toportions of the environment not covered by visual sensors, and overlapareas correspond to portions of the environment covered by multiplevisual sensors. Method 800 ends following block 835, or optionally block845.

FIG. 9 illustrates a method of producing combined image information toaddress deadspace areas in the environment, according to one embodiment.In one embodiment, method 900 may be performed following block 845 ofmethod 800.

Method 900 begins at block 905, where one or more deadspace areas aredetermined within the environment. In one embodiment, deadspace areasare determined based on the image information acquired from thearrangement of visual sensors disposed throughout the environment. Atblock 910, it is determined whether a field of view of a person overlapsat least part of a deadspace area. The image information used for thefield of view determination may be acquired from a visual sensor of apersonal computing device.

If the field of view overlaps (“YES”), the method proceeds to block 915,where the image information is stored by the second (environmental)computing device. Thus, the image information acquired using thepersonal computing device may subsequently be included with the combinedimage information.

If the field of view does not overlap (“NO”), the method proceeds toblock 925 and/or block 935. At block 925, depending on the particulararrangement, the second visual sensors may be reoriented and/ordisplaced to acquire image information corresponding to the deadspacearea. At block 935, the second (environmental) computing device promptsa person to acquire image information for the deadspace areas using avisual sensor included in the associated personal computing device.

FIG. 10 illustrates a method of producing combined image informationbased on overlap areas in the environment, according to one embodiment.In one embodiment, method 1000 may be performed following block 845 ofmethod 800.

Method 1000 begins at block 1005, where one or more overlap areas aredetermined corresponding to two or more second visual sensors. In oneembodiment, overlap areas are determined based on the image informationacquired from the arrangement of visual sensors disposed throughout theenvironment.

At block 1015, image information for each of the two or more“overlapping” second visual sensors is analyzed to determine respectivenumbers of identified items included therein. In one embodiment, thenumbers of identified items corresponds specifically to the overlaparea. In another embodiment, the numbers of identified items includesall of the image information.

At block 1025, the numbers of identified items are compared with an itemthreshold value. If the threshold is not met (“NO”), the method proceedsto block 1025, where first image information is stored by the secondcomputing device. In some embodiments, block 1025 may include, prior tostoring the first image information, prompting a person to acquire thefirst image information using a visual sensor included in the associatedpersonal computing device.

If the threshold is met (“YES”), the method proceeds to block 1035.Generally, meeting the threshold value may indicate that the imageinformation is of suitable quality for use in the combined imageinformation. At block 1035, the image information corresponding to thelargest number of identified items is assigned as the primary imageinformation for the overlap area. At block 1045, the other imageinformation (which have smaller numbers of identified items) is assignedas auxiliary image information for the overlap area.

The descriptions of the various embodiments of the present disclosurehave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

In the preceding, reference is made to embodiments presented in thisdisclosure. However, the scope of the present disclosure is not limitedto specific described embodiments. Instead, any combination of thefollowing features and elements, whether related to differentembodiments or not, is contemplated to implement and practicecontemplated embodiments. Furthermore, although embodiments disclosedherein may achieve advantages over other possible solutions or over theprior art, whether or not a particular advantage is achieved by a givenembodiment is not limiting of the scope of the present disclosure. Thus,the following aspects, features, embodiments and advantages are merelyillustrative and are not considered elements or limitations of theappended claims except where explicitly recited in a claim(s). Likewise,reference to “the invention” shall not be construed as a generalizationof any inventive subject matter disclosed herein and shall not beconsidered to be an element or limitation of the appended claims exceptwhere explicitly recited in a claim(s).

Aspects of the present disclosure may take the form of an entirelyhardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module,” or “system.”

The present disclosure may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent disclosure.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present disclosure may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Embodiments of the disclosure may be provided to end users through acloud computing infrastructure. Cloud computing generally refers to theprovision of scalable computing resources as a service over a network.More formally, cloud computing may be defined as a computing capabilitythat provides an abstraction between the computing resource and itsunderlying technical architecture (e.g., servers, storage, networks),enabling convenient, on-demand network access to a shared pool ofconfigurable computing resources that can be rapidly provisioned andreleased with minimal management effort or service provider interaction.Thus, cloud computing allows a user to access virtual computingresources (e.g., storage, data, applications, and even completevirtualized computing systems) in “the cloud,” without regard for theunderlying physical systems (or locations of those systems) used toprovide the computing resources.

Typically, cloud computing resources are provided to a user on apay-per-use basis, where users are charged only for the computingresources actually used (e.g., an amount of storage space consumed by auser or a number of virtualized systems instantiated by the user). Auser can access any of the resources that reside in the cloud at anytime, and from anywhere across the Internet. In context of the presentdisclosure, a user may access applications (e.g., a retail store app fora mobile computing device) or related data (e.g., compiled shoppingdata) available in the cloud. Doing so allows a user to access thisinformation from any computing system attached to a network connected tothe cloud (e.g., the Internet).

While the foregoing is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

What is claimed is:
 1. A computer-implemented method to provide anextended vision within an environment having a plurality of items, theextended vision based on a field of view of a person having a firstcomputing device coupled with a first visual sensor within theenvironment, the environment associated with at least a second computingdevice coupled with at least a second visual sensor disposed within theenvironment, at least one of the first and second computing devicescoupled with a display device and including a user interface (UI) fortraversing the extended vision, the method comprising: analyzing firstimage information acquired using the first visual sensor to determinethe field of view of the person, the first image information includingone or more first items of the plurality of items; analyzing secondimage information acquired using the second visual sensor, the secondimage information including one or more second items of the plurality ofitems; based on the determined field of view, associating the firstimage information with the second image information to produce combinedimage information; and based on input provided through the UI,displaying via the display device at least portions of the combinedimage information to provide the extended vision.
 2. The method of claim1, wherein associating the first and second image information includesstitching the first and second image information.
 3. The method of claim1, wherein the combined image information is navigable via the displaydevice.
 4. The method of claim 1, wherein the first computing device isa body-worn computing device and the first visual sensor is aforward-looking camera integrated with the first computing device. 5.The method of claim 1, wherein the first computing device is a handheldcomputing device and the person manipulates an orientation of thehandheld computing device to acquire the first image information.
 6. Themethod of claim 1, wherein the at least a second visual sensor comprisesa plurality of second visual sensors, the method further comprising:determining that an arrangement of the plurality of second visualsensors results in at least a first deadspace area representing aportion of the environment not acquired in the image information of anyof the plurality of second visual sensors; and upon determining that thefield of view of the person overlaps at least part of the firstdeadspace area, storing the associated first image information using thesecond computing device.
 7. The method of claim 1, wherein the at leasta second visual sensor comprises a plurality of second visual sensors,the method further comprising: determining that an arrangement of theplurality of second visual sensors results in at least a first overlaparea representing a portion of the environment acquired in the imageinformation of two or more of the plurality of second visual sensors;and analyzing, for each of the two or more second visual sensorscorresponding to the first overlap area, corresponding image informationto identify a respective number of the plurality of items; assigning,based on a largest of the numbers, the corresponding visual sensor ofthe two or more second visual sensors as a master view for the firstoverlap area; and assigning other visual sensors of the two or moresecond visual sensors as an auxiliary view for the first overlap area.8. The method of claim 6, wherein none of the numbers of identifieditems meet a predetermined threshold value, the method furthercomprising: upon determining that the field of view of the personoverlaps at least part of the first overlap area, storing the associatedfirst image information using the second computing device.
 9. The methodof claim 1, further comprising: performing, in response to the inputprovided through the UI, at least one of reorienting and displacing thesecond visual sensor.
 10. A computer program product to provide anextended vision within an environment having a plurality of items, theextended vision based on a field of view of a person having a firstcomputing device coupled with a first visual sensor within theenvironment, the environment associated with at least a second computingdevice coupled with at least a second visual sensor disposed within theenvironment, at least one of the first and second computing devicescoupled with a display device and including a user interface (UI) fortraversing the extended vision, the computer program product comprising:a computer-readable storage medium having computer-readable program codeembodied therewith, the computer-readable program code executable by oneor more computer processors to perform an operation that includes:analyzing first image information acquired using the first visual sensorto determine the field of view of the person, the first imageinformation including one or more first items of the plurality of items;analyzing second image information acquired using the second visualsensor, the second image information including one or more second itemsof the plurality of items; based on the determined field of view,associating the first image information with the second imageinformation to produce combined image information; and based on inputprovided through the UI, displaying via the display device at leastportions of the combined image information to provide the extendedvision.
 11. The computer program product of claim 10, whereinassociating the first and second image information includes stitchingthe first and second image information.
 12. The computer program productof claim 10, wherein the combined image information is navigable via thedisplay device.
 13. The computer program product of claim 10, whereinthe at least a second visual sensor comprises a plurality of secondvisual sensors, the operation further comprising: determining that anarrangement of the plurality of second visual sensors results in atleast a first deadspace area representing a portion of the environmentnot acquired in the image information of any of the plurality of secondvisual sensors; and upon determining that the field of view of theperson overlaps at least part of the first deadspace area, storing theassociated first image information using the second computing device.14. The computer program product of claim 10, wherein the at least asecond visual sensor comprises a plurality of second visual sensors, theoperation further comprising: determining that an arrangement of theplurality of second visual sensors results in at least a first overlaparea representing a portion of the environment acquired in the imageinformation of two or more of the plurality of second visual sensors;and analyzing, for each of the two or more second visual sensorscorresponding to the first overlap area, corresponding image informationto identify a respective number of the plurality of items; assigning,based on a largest of the numbers, the corresponding visual sensor ofthe two or more second visual sensors as a master view for the firstoverlap area; and assigning other visual sensors of the two or moresecond visual sensors as an auxiliary view for the first overlap area.15. The computer program product of claim 14, wherein none of thenumbers of identified items meet a predetermined threshold value, theoperation further comprising: upon determining that the field of view ofthe person overlaps at least part of the first overlap area, storing theassociated first image information using the second computing device.16. The computer program product of claim 10, the operation furthercomprising: performing, in response to the input provided through theUI, at least one of reorienting and displacing the second visual sensor.17. A system to provide an extended vision within an environment havinga plurality of items, the extended vision based on a determined field ofview of a person and traversable using a user interface (UI), the systemcomprising: a plurality of computer processors, at least one processorincluded in a first computing device associated with the person, and atleast one other processor included in at least one second computingdevice associated with the environment and communicatively coupled withthe first computing device; a first visual sensor communicativelycoupled with the first computing device, the first visual sensordisposed within the environment and configured to acquire first imageinformation; one or more second visual sensors that are communicativelycoupled with the second computing device, the second visual sensorsdisposed within the environment and configured to acquire second imageinformation; and a display device communicatively coupled with at leastone of the first computing device and the second computing device;wherein the plurality of computer processors are configured to performan operation that includes: analyzing the first image information todetermine the field of view of the person, the first image informationincluding one or more identified first items of the plurality of items;analyzing the second image information, the second image informationincluding one or more identified second items of the plurality of items;based on the determined field of view, associating the first imageinformation with the second image information to produce combined imageinformation; and based on input provided through the UI, displaying viathe display device at least portions of the combined image informationto provide the extended vision.
 18. The system of claim 17, wherein thefirst computing device is a body-worn computing device and the firstvisual sensor is a forward-looking camera integrated with the firstcomputing device.
 19. The system of claim 17, wherein the firstcomputing device is a handheld computing device and the personmanipulates an orientation of the handheld computing device to acquirethe first image information.
 20. The system of claim 17, wherein thecombined image information is navigable via the display device.